Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docs(ansible): add readme with manual artifacts downloading explanation #3850

Merged
merged 15 commits into from
Sep 26, 2023
Merged

docs(ansible): add readme with manual artifacts downloading explanation #3850

merged 15 commits into from
Sep 26, 2023

Conversation

lexavtanke
Copy link
Contributor

@lexavtanke lexavtanke commented Sep 20, 2023 •
edited by esteve
Loading

Description

add README to ansible role artifacts. It provide instruction how to download package artifacts manually. Also link to this README will be used in documentation on pages:

See autowarefoundation/autoware.universe#3137

Tests performed

Not applicable.

Effects on system behavior

Not applicable.

Pre-review checklist for the PR author

The PR author must check the checkboxes below when creating the PR.

In-review checklist for the PR reviewers

The PR reviewers must check the checkboxes below before approval.

Post-review checklist for the PR author

The PR author must check the checkboxes below before merging.

  • There are no open discussions or they are tracked via tickets.

After all checkboxes are checked, anyone who has write access can merge the PR.

@lexavtanke lexavtanke requested a review from xmfcx September 20, 2023 14:07
@lexavtanke lexavtanke changed the title doc(ansible): add readme with manual artifacts downloading explanation docs(ansible): add readme with manual artifacts downloading explanation Sep 20, 2023
@lexavtanke lexavtanke requested a review from esteve September 20, 2023 14:56
esteve
esteve reviewed Sep 20, 2023
View reviewed changes
Copy link
Contributor

@esteve esteve left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lexavtanke LGTM. What do you think of also providing a SHA256SUMS file so that the downloaded files can be tested against their checksums?

@lexavtanke
Copy link
Contributor Author

@esteve Yes, make sense. I will switch it to draft for now and update.

@lexavtanke lexavtanke marked this pull request as draft September 20, 2023 16:17
@esteve esteve mentioned this pull request Sep 21, 2023
Closed
38 tasks
@lexavtanke lexavtanke marked this pull request as ready for review September 21, 2023 11:01
@lexavtanke lexavtanke requested a review from esteve September 22, 2023 12:18
@esteve
Copy link
Contributor

esteve commented Sep 22, 2023

@lexavtanke sorry I wasn't more specific. I meant adding a checksum file like this:

https://help.ubuntu.com/community/HowToSHA256SUM

so that users can check the files easily by running sha256sum -c SHA256SUMS, we might want to sign that file with GPG eventually as well.

@lexavtanke
Copy link
Contributor Author

@esteve before I had started I also had this Idea because I wasn't sure. But I come to a conclusion that it will work for the first time and if user will update models manually one by one and will keep engines in the folders and run chechsum it can be confusing. After initial run of autoware user will have some additional files like trt engines.
Here is the example for lidar_centerpoint.
image

Do you think it will be better to push user to update all artifacts simultaneously? If so we will need to add a new playbook to ansible and modify setup-dev-env.sh or provide separate script for updating.

@esteve
Copy link
Contributor

esteve commented Sep 22, 2023

@lexavtanke I think it's fine if the models change because the user modifies them, but the first time they are downloaded it'd be useful for the user if they can verify the files easily (with sha2156 for example). Here's the SHA256SUMS file I generated locally:

7fe62fcebe0e0f62a000d06aa94d779feb444d933671a4a3189fe01be8c19a00  ./image_projection_based_fusion/pts_backbone_neck_head_pointpainting.onnx
3ca452ea5ca9467bf782955f75704ba8466841e275e8b8acd991b9911d53249e  ./image_projection_based_fusion/pts_voxel_encoder_pointpainting.onnx
86348d8c4bced750f54288b01cc471c0d4f1ec9c693466169ef19413731e6ecc  ./lidar_apollo_instance_segmentation/hdl-64.onnx
eec521ebad7553d0ea2c90472a293aecb7499ab592632f0e100481c8196eb421  ./lidar_apollo_instance_segmentation/vlp-16.onnx
95ef950bb694bd6de91b7e47f5d191d557e92a7f5e2a6bdf655a8b5eed4075cc  ./lidar_apollo_instance_segmentation/vls-128.onnx
3fe7e128955646740c41a25be0c8f141d5a94594fe79d7405fe2a859e391542e  ./lidar_centerpoint/pts_backbone_neck_head_centerpoint.onnx
9bb0b634f3664bd098ce7d6a3d8a9fb7cc8d9b8252b27f302c71e43316bab551  ./lidar_centerpoint/pts_backbone_neck_head_centerpoint_tiny.onnx
dc1a876580d86ee7a341d543f8ade2ede7f43bd032dc5b44155b1f0175405764  ./lidar_centerpoint/pts_voxel_encoder_centerpoint.onnx
2c53465715c1fd2e9dc5727ef3fca74f4cdf0538f74286b0946e219d0ca5693b  ./lidar_centerpoint/pts_voxel_encoder_centerpoint_tiny.onnx
634a1132eb33f8091d60f2c346ababe8b905ae08387037aed883953b7329af84  ./tensorrt_yolo/coco.names
61e922f76918dd3d8e0abdc5fb7406f390609e08bd8ab9e5d3b97afb00f30f8c  ./tensorrt_yolo/yolov3.onnx
0e877c716fbf8a2b431ee3e57f6c7411a6741319b52c32c6dafc53c7e1b17027  ./tensorrt_yolo/yolov4-tiny.onnx
7c7343156c1bd4b397fd1e44b27334691a6219db3ce2e29a03b72af65ddb8f39  ./tensorrt_yolo/yolov4.onnx
a627e5f70180a8746482b572194090466db62c8d1256602c1cd20374dd960e34  ./tensorrt_yolo/yolov5l.onnx
ee6f67f7c00a34cc4cef2fdd9db30dd714df1a4fb2d7e9fc1731cfe85b673133  ./tensorrt_yolo/yolov5m.onnx
be335ff7746957debf1a6903a61fa3f568b780b4afe4958edf2d4bc98e9e0825  ./tensorrt_yolo/yolov5s.onnx
d7cb4cd7078f87bda22a37828d72867accecedf9f74d0d87b5cc1f6f1180a019  ./tensorrt_yolo/yolov5x.onnx
3540a365bfd6d8afb1b5d8df4ec47f82cb984760d3270c9b41dbbb3422d09a0c  ./tensorrt_yolox/label.txt
cc378d327db5616b0b3a4d077bf37100c25a50ecd22d2b542f54098da100f34c  ./tensorrt_yolox/yolox-sPlus-T4-960x960-pseudo-finetune.EntropyV2-calibration.table
f5054e8a890c3be86dc1b4b89a5a36fb2279d4f6110b0159e793be062641bf65  ./tensorrt_yolox/yolox-sPlus-T4-960x960-pseudo-finetune.onnx
b9e9d7da33342262ccaea4469b4d02b8abb32b6d7bf737f9e0883fece1b8f580  ./tensorrt_yolox/yolox-sPlus-opt.EntropyV2-calibration.table
36b0832177b01e6b278e00c7369f1de71e616c36261cbae50f0753d41289da01  ./tensorrt_yolox/yolox-sPlus-opt.onnx
471a665f4243e654dff62578394e508db22ee29fe65d9e389dfc3b0f2dee1255  ./tensorrt_yolox/yolox-tiny.onnx
1a5a49eeec5593963eab8d70f48b8a01bfb07e753e9688eb1510ad26e803579d  ./traffic_light_classifier/lamp_labels.txt
55ebb0d117a5e8943f8d1c6769f1d856b533079d4d871d8e923255cc992ad48a  ./traffic_light_classifier/traffic_light_classifier_efficientNet_b1_batch_1.onnx
684e29843e3128eadb774018730644b3ab9b0a06dc4cdaeed579c2f3fa5d5265  ./traffic_light_classifier/traffic_light_classifier_efficientNet_b1_batch_4.onnx
44d94540fa8b89dfb39cd9a8523cf010ddfb10ea2f1f9b53bf3618ce7f4912ad  ./traffic_light_classifier/traffic_light_classifier_efficientNet_b1_batch_6.onnx
455b71b3b20d3a96aa0e49f32714ba50421f668a2f9b9907c30b1346ac8a3703  ./traffic_light_classifier/traffic_light_classifier_mobilenetv2_batch_1.onnx
41bb79a23a4ac57956adb8e9cb3904420db1b0cd032e97b670cc4f8b174ae3fe  ./traffic_light_classifier/traffic_light_classifier_mobilenetv2_batch_4.onnx
e4792eed6a46fdbd02be2f3a4f1ce91f36fa77698493caf3102e445178c0f058  ./traffic_light_classifier/traffic_light_classifier_mobilenetv2_batch_6.onnx
a41e6e3324e32c30b3b2fe38908eaf3471e2bfdaeb9e14ca0c1c3bc0275119c6  ./traffic_light_fine_detector/tlr_labels.txt
922839fcf22bd32ae5065146fcec193e9d6360ca03bd4c83faea835045daf8eb  ./traffic_light_fine_detector/tlr_yolox_s_batch_1.onnx
b3c6e00acc6ff547d165469684ffb620a9a6330e9d591d445f50c4cf5cb4e292  ./traffic_light_fine_detector/tlr_yolox_s_batch_4.onnx
2824d4c5b7ab5f6bfd41e43e82747107c53e1c727b1cf1dd6746bc49e6749128  ./traffic_light_fine_detector/tlr_yolox_s_batch_6.onnx
e29e6ee68751a270fb285fd037713939ca7f61a897b4c3a7ab22b0d6a9a21ddf  ./traffic_light_ssd_fine_detector/mb2-ssd-lite-tlr.onnx
a41e6e3324e32c30b3b2fe38908eaf3471e2bfdaeb9e14ca0c1c3bc0275119c6  ./traffic_light_ssd_fine_detector/voc_labels_tl.txt
1f660e15f95074bade32b1f80dbf618e9cee1f0b9f76d3f4671cb9be7f56eb3a  ./yabloc_pose_initializer/resources.tar.gz

I'm going to use it to update the ansible scripts so that all the files have a SHA256 sum (right now, some still a have an MD5 one)

@lexavtanke
Copy link
Contributor Author

@esteve I've added SHA256SUM to README.

@esteve
Copy link
Contributor

esteve commented Sep 26, 2023

@lexavtanke I see, I meant a separate file, so that users can run sha256sum -c SHA256SUMS to check the integrity of the files. If they have to copy and paste the contents of README, they won't do it and it's also more difficult to automate.

esteve
esteve requested changes Sep 26, 2023
View reviewed changes
Copy link
Contributor

@esteve esteve left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The SHA256SUMS file can live in the same folder as the README, users can either download it so that it's in ~/autoware_data/SHA256SUMS or we can tell them to issue the following command:

$ cd ~/autoware_data/
$ wget -q -O - https://raw.githubusercontent.com/autowarefoundation/autoware/main/ansible/roles/artifacts/SHA256SUMS | sha256sum -c

@lexavtanke lexavtanke requested a review from esteve September 26, 2023 11:25
esteve
esteve approved these changes Sep 26, 2023
View reviewed changes
Copy link
Contributor

@esteve esteve left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lexavtanke thank you so much for addressing my feedback 🙂

@esteve esteve enabled auto-merge (squash) September 26, 2023 11:35
@esteve esteve merged commit 459dda9 into autowarefoundation:main Sep 26, 2023
14 checks passed
@tier4-autoware-public-bot tier4-autoware-public-bot bot mentioned this pull request Sep 27, 2023
Open
@lexavtanke lexavtanke deleted the add-readme-to-artifacts-ansible-role branch October 18, 2023 08:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@xmfcx xmfcx xmfcx approved these changes

@esteve esteve esteve approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@lexavtanke @esteve @xmfcx